[ Top | Up | Prev | Next | Map | Index ]

Readme for analog3.1

Inclusions and exclusions

After aliasing each item, analog decides whether that item is wanted or not. The whole line is only counted if all the items are wanted. (If one type of item doesn't occur on a line, that item's counted as wanted on that line.) Whether an item is wanted or not is determined by INCLUDE and EXCLUDE commands specified by the user. These commands can be used to exclude requests from your local users, for example, or to analyse only files in a subdirectory.

The rule for determining whether an item is included or excluded is as follows. All the INCLUDE and EXCLUDE commands for that item are considered one by one in order, and the item is included or excluded according to the last command it matched. Items which don't match any of the INCLUDE or EXCLUDE commands are included if the first command was an exclusion, and excluded if the first command was an inclusion. For example, the configuration

FILEINCLUDE /~sret1/*
FILEEXCLUDE /~sret1/backgammon/*,/~sret1/analog/*
FILEINCLUDE /~sret1/backgammon/*.gif

would instruct the program to examine only my files, excluding my backgammon and analog files, but including gifs in my backgammon directory. On the other hand,

FILEEXCLUDE /~sret1/*/img/*

would analyse all files, except for images in my various directories. Note that inclusions and exclusions can contain any number of wildcards.

The relevant commands for the other types of item are HOSTINCLUDE and HOSTEXCLUDE; BROWINCLUDE and BROWEXCLUDE; REFINCLUDE and REFEXCLUDE; USERINCLUDE and USEREXCLUDE; and VHOSTINCLUDE and VHOSTEXCLUDE. If you get confused with all the inclusions and exclusions, remember that you can always run analog -settings to see what the options you have specified represent.

There is also one other pair of commands which belongs in this category, namely the FROM and TO commands. These specify a time period to restrict the analysis to. The simplest usage of these commands is FROM yyMMdd or FROM yyMMdd:hhmm, where yy represents the last two digits of the year (analog assumes that the year is between 1970 and 2069), MM represents the month, dd is the date, hh the hour, and mm the minute. So, for example, to analyse only requests from July 1999 to June 2000 I would use the configuration

FROM 990701
TO   000630

Alternatively, each of the components can be preceded by + or - to represent time relative to the time at which the program was invoked. In this case, the date can have more than 2 digits. This allows constructions like

FROM -01-00+01   # from tomorrow last year
TO -00-0131  # to the end of last month (OK even if last month
             # didn't have 31 days)
FROM -00-00-112
TO   -00-00-01  # statistics for the last 16 weeks
FROM -00-00-00:-06+01  # statistics for the last 6 hours

There are command line abbreviations +F and +T for the FROM and TO commands; for example, +T-00-00-01:1800 looks at statistics until 6pm yesterday. -F and -T turn off the from and to, as do FROM OFF and TO OFF.

There are also INCLUDE and EXCLUDE commands for most of the reports. These exclude individual lines from particular reports. So, for example, the command

REFREPEXCLUDE http://www.yahoo.com/*

would exclude Yahoo! referrers from the referrer report. However, it would not exclude them from the failed referrer report, the referring site report, etc. (you need to use FAILREFEXCLUDE, REFSITEEXCLUDE etc. for that); nor would it prevent other analysis of logfile lines with those referrers, as REFEXCLUDE would. Also REFREPEXCLUDE would include the referrers in the "not listed" line at the bottom of the report.

The full list of these commands is REQINCLUDE and REQEXCLUDE; REDIRINCLUDE and REDIREXCLUDE; FAILINCLUDE and FAILEXCLUDE; TYPEINCLUDE and TYPEEXCLUDE; DIRINCLUDE and DIREXCLUDE; HOSTREPINCLUDE and HOSTREPEXCLUDE; DOMINCLUDE and DOMEXCLUDE; REFREPINCLUDE and REFREPEXCLUDE; REFSITEINCLUDE and REFSITEEXCLUDE; REDIRREFINCLUDE and REDIRREFEXCLUDE; FAILREFINCLUDE and FAILREFEXCLUDE; BROWSUMINCLUDE and BROWSUMEXCLUDE; FULLBROWINCLUDE and FULLBROWEXCLUDE; VHOSTREPINCLUDE and VHOSTREPEXCLUDE; USERREPINCLUDE and USERREPEXCLUDE; and FAILUSERINCLUDE and FAILUSEREXCLUDE. The inclusion or exclusion applies to the unaliased name, if you are doing any output aliases.

You can also use the symbolic word pages in suitable INCLUDE and EXCLUDE commands; one very common command is

REQINCLUDE pages

to include only pages in the request report.

Analog determines which files should count as pages (and thus which requests count as page requests) using another INCLUDE/EXCLUDE pair, called PAGEINCLUDE and PAGEEXCLUDE. By default, *.html, *.htm and directories (*/) count as pages. But you change the list by commands like

PAGEINCLUDE *.ps,*.ps.gz
PAGEEXCLUDE sret1.html

(I.e., Postscript and gzipped Postscript are pages, but sret1.html isn't).

There are a couple more INCLUDE and EXCLUDE commands which I'll mention now while we're on the subject. In the Request Report and the three referrer reports (Referrer Report, Redirected Referrer Report and Failed Referrer Report), analog can link to the files which it's listing. There are commands LINKINCLUDE and LINKEXCLUDE for the Request Report, and REFLINKINCLUDE and REFLINKEXCLUDE for the referrer reports, to specify exactly which files are linked to. (By default you get all pages). So, for example, LINKINCLUDE *.txt would link to *.txt files as well as pages in the Request Report, and REFLINKEXCLUDE * would tell analog to make no links in the three referrer reports.

Finally, there are commands called ARGSINCLUDE and ARGSEXCLUDE, and REFARGSINCLUDE and REFARGSEXCLUDE. Sometimes a URL contains arguments after a question mark. For example, the URL

/cgi-bin/script.pl?x=1&y=2

runs the /cgi-bin/script.pl program with arguments x=1 and y=2. (Sometimes the server records the arguments in a separate field in the logfile, but if so you can use the %q field in the LOGFORMAT command, and analog will translate the filename to the above format).

Analog can either read or ignore the arguments. If the command ARGSEXCLUDE /cgi-bin/script.pl were given, analog would ignore the arguments to that file, and so treat the above URL as being the same as /cgi-bin/script.pl. On the other hand, if ARGSINCLUDE /cgi-bin/script.pl were specified, analog would read the arguments, and treat the above URL as a different file from /cgi-bin/script.pl (or from /cgi-bin/script.pl?y=2&x=1), although a grand total for /cgi-bin/script.pl would still be listed in the Request Report.

REFARGSINCLUDE and REFARGSEXCLUDE are the same for referrers. By default, all arguments are included. The check for whether the arguments should be included happens before the filename is aliased: this means that you can't use pages in this command, because we don't know whether a file is a page until after it's been aliased.

If you want to see the arguments in a report you may also have to set the appropriate ARGSFLOOR command.

Stephen Turner
E-mail: sret1@cam.ac.uk

[ Top | Up | Prev | Next | Map | Index ]